The present invention relates to image-capturing apparatuses that capture natural images and that detect optical signals indicating various types of information.
Currently, image-capturing devices, for example, charge-coupled device (CCD) image sensors and complementary metal-oxide semiconductor (CMOS) image sensors, are available at low prices. Thus, many home electrical appliances and information technology (IT) apparatuses, for example, camcorders, digital still cameras, mobile phones, and personal computers, include cameras. Since CMOS image sensors and other general metal-oxide semiconductor (MOS) devices can be manufactured on a common production line, an image-sensing unit and other system units such as a signal-processing unit and an analog-to-digital converter (ADC) can be readily mounted on the same chip.
Moreover, an image sensor, disclosed in Japanese Unexamined Patent Application Publication No. 2003-169251, not only captures images, but also carries out other processes by analog and digital calculations.
This image sensor will now be described. FIG. 9 is a block diagram illustrating the structure of the CMOS image sensor. This image sensor can capture normal images that are referred to as natural images and three-dimensional range data of objects in images. The image sensor includes a pixel array 10 that has a two-dimensional array of pixels detecting light, a current-to-voltage (I-V) conversion circuit 12 that converts current signals detected by the pixel array 10 to voltage signals, a correlated double sampling (CDS) circuit 14 that filters out noise in image signals, an analog memory array 16 that holds the pixel signals detected by the pixel array 10, a current mirror 18 that outputs the pixel signals detected by the pixel array 10 to the analog memory array 16, a comparator-latch unit 20 that calculates the difference among values in memory cells in the analog memory array 16 and that latches the resulting difference value, a vertical (V) pixel scanner 22, a horizontal (H) pixel scanner 24, a vertical (V) memory scanner 26, and a horizontal (H) memory scanner 28. The V pixel scanner 22 and the H pixel scanner 24 control scanning of the pixel array 10. The V memory scanner 26 and the H memory scanner 28 control scanning of the analog memory array 16.
FIG. 10 is a block diagram illustrating the connection of the circuits shown in FIG. 9. FIG. 11 is a block diagram illustrating switching operation in the pixel array 10 shown in FIG. 10. FIG. 12 is a circuit diagram of one pixel in the pixel array 10 shown in FIG. 9.
As shown in FIG. 10, the pixel array 10 includes four types of pixels 30, i.e., yellow (Ye), cyan (Cy), green (G), and magenta (Mg). The analog memory array 16 includes memory units F1 to F4 corresponding to four respective frames in high-speed frame scanning. Each of the memory units F1 to F4 includes memory cells 32. Signals from the pixels 30 are read through vertical signal lines 34 extending through the pixel array 10. Each vertical signal line 34 includes a switch S11 at the upper portion and a switch S12 at the lower portion. The switches S11 and S12 are turned on and off in response to a read operation of pixel signals. A switch S13 is provided between the analog memory array 16 and the comparator-latch unit 20 and is turned on and off in response to a read operation of memory signals.
As shown in FIG. 12, in this image sensor, five MOS transistors are provided for one pixel. Each pixel includes a photodiode PD serving as a photoelectric transducer, a floating diffusion part FD, a transfer transistor T11 that transfers signal charge generated at the photodiode PD to the floating diffusion part FD, an amplifying transistor T12 that outputs voltage signals or current signals based on the signal charge transferred to the floating diffusion part FD, a reset transistor T13 that resets the floating diffusion part FD to a power source potential based on reset signals (RST), a transfer-controlling transistor T14 that controls timing for switching the transferring transistor T11 based on column selection signals (CGm) and charge transfer signals (TX), and a selecting transistor T15 that controls timing for the amplifying transistor T12 to output signals based on pixel selection signals (SEL).
When normal image data is read out, signals from the pixels 30 are read out through the vertical signal lines 34 in the upward direction. The V pixel scanner 22 and the H pixel scanner 24 sequentially scan each row and each column to read out the signals from the pixels 30. Then, the signals from the pixels 30 are processed in the I-V conversion circuit 12 and the CDS circuit 14 and are amplified to be output outside the chip as analog image signals.
On the other hand, when three-dimensional range data is processed, signals from the pixels 30 are read out through the vertical signal lines 34 in the downward direction. In processing three-dimensional range data, frame scanning is carried out at a high rate of, for example, 14 kfps while a slit-shaped infrared light beam is emitted to an object and the reflected light is detected. Then, the difference among four consecutive frames is calculated.
As shown in FIG. 11, in a light-detecting section of the image sensor, color filters are provided on the pixels 30. RGBG primary-color filters or CMYG complementary-color filters are used. In this image sensor, since color filters need to transmit infrared light when three-dimensional range data is processed, CMYG complementary-color filters having high transmittance of near-infrared light are used. When three-dimensional range data is processed, four pixels corresponding to CMYG are read out as one range operating unit (ROU) to be combined in order to cancel differences in transmittance of near-infrared light in the CMYG filters. The analog memory array 16 includes the four memory units F1 to F4 for holding signals of four consecutive frames. In each of the memory units F1 to F4, the memory cells 32 are provided corresponding to respective ROUs in the pixel array 10. In this arrangement, the signals from the pixels 30 pass through the current mirror 18 and are temporarily held in the memory cells 32 for four consecutive frames. Then, the comparator-latch unit 20 calculates the difference between combined signals from two leading frames and combined signals from two succeeding frames and latches the resulting difference as binary data. When the ROUs detect the infrared light beam, the calculated difference is “1”, and this data is output outside the image sensor.
In processing three-dimensional range data, the timing of detecting the infrared light beam can be used for measuring the distance between each pixel and the corresponding object.
Data communication can be carried out with signals obtained by encoding patterns (light intensity change) of a blinking light-emitting diode (LED), using the same image sensor as described above.
For example, as shown in FIG. 13, an LED light source (an LED beam controller) 2 blinks in the visual field of a camera 1, and ID data is generated by encoding patterns of blinking light. As in processing three-dimensional range data, the image sensor is controlled so as to calculate the difference among four consecutive frames. Each ROU detects timing of changes in the LED light, and outputs this data outside the image sensor. An external device derives the patterns of the blinking LED from this timing data. Thus, the external device can obtain data on IDs and pixels that detected the blinking LED light, and thus can identify objects in an image, superimpose the ID data, other data related to the ID data, and the objects on a display, and capture motions of the objects.
The known image sensor described above can capture image data, and can detect a slit-shaped infrared light beam for processing three-dimensional range data or can detect blinking LED light. However, since the same pixels detect light for these functions using common signals lines in the known image sensor, the operation of outputting image data and the operation of processing three-dimensional range data or of detecting blinking LED light cannot be simultaneously carried out.
Thus, in the known image sensor, the operation mode must change between an image-capturing mode and an optical-change-detecting mode every frame so that more than one type of data seem to be simultaneously output.
However, in the known image sensor, when image data is processed, image data is captured every other frame, and thus the usability of the image sensor is impaired. For example, in a system that is designed so as to use consecutive frames captured by a regular image sensor, the known image sensor may be installed instead of the regular image sensor so as to carry out a three-dimensional range data-processing function and an ID data communicating function in addition to an image-capturing function. In this case, there is no compatibility of image data between the known image sensor and the regular image sensor. Thus, the system needs to be rebuilt so that the system can control image data captured by the known image sensor.
Moreover, since every other frame is available in detecting blinking LED light, when ID data is retrieved from an LED provided in an object that moves quickly, the object may not be correctly tracked due to time lag.